Driver alertness monitoring system

ABSTRACT

A method for monitoring driver alertness in a vehicle is disclosed. Image data representing images of an eye of a driver are obtained, the images having been captured while the driver is operating the vehicle. An eye openness measure of the eye of the driver is computed, based on the image data. A determination is made that the driver is in an unsafe state for driving based on the eye openness measure, and an alert is generated, while the driver is operating the vehicle, based on determining that the driver is in the unsafe state.

FIELD

At least one embodiment of the present disclosure pertains to in-vehicleprocessing systems, and more particularly, to an in-vehicle driveralertness monitoring system.

BACKGROUND

Driver drowsiness, fatigue and distraction while driving are majorsources of automobile accidents. According to the National HighwayTraffic Safety Administration (NHTSA), about 100,000 police-reportedcrashes are the direct result of driver fatigue resulting in injuries,loss of life and $12.5 billion financial loss. In 2015, there were 824fatalities that were related to drowsy driving.

There are many causes for driver drowsiness and distraction. Some of thesignificant contributors are medication, sleep disorder or poor night'srest and mobile devices. The European New Car Assessment Programme (EuroNCAP), the voluntary vehicle safety rating system, recently introduced“occupant status monitoring” rating as part of their 5-star ratingcategories. This rating requires the vehicle to monitor the alertnessand the attention level of the driver. Automotive original equipmentmanufacturers and fleets are looking for innovative driver monitoringsolutions that are cost effective and robust for automotive environment.

BRIEF DESCRIPTION OF THE DRAWINGS

One or more embodiments of the present disclosure are illustrated by wayof example and not limitation in the figures of the accompanyingdrawings, in which like references indicate similar elements.

FIG. 1 is a high-level block diagram of an example of the physicalstructure of an in-vehicle driver alertness monitoring system.

FIG. 2 is a block diagram of an example of the functional elements ofthe in-vehicle driver alertness monitoring system of FIG. 1 .

FIG. 3 is a flowchart showing an example of the overall processperformed by the Head Orientation and Feature Landmarks DetectionModule.

FIG. 4 shows an example of facial landmarks on the face of a person.

FIG. 5 shows an example of landmarks representing the eyelid contoursand center of the pupil of the eyes of a driver.

FIG. 6 is a flowchart showing an example of the process performed by theDriver Behavior Data Analysis Module.

FIG. 7A schematically shows eyelid contours and landmarks of an eye inaccordance with a first eye openness measure (EOM) calculation method.

FIG. 7B schematically shows how a triangle may be defined by landmarksin the first EOM calculation method.

FIG. 8 schematically shows eyelid contours and landmarks of an eye inaccordance with an alternate EOM calculation method.

FIG. 9 is a flowchart showing an example of the process of the first EOMcalculation method and corresponding to FIGS. 7A and 7B.

FIG. 10 is a flowchart showing an example of the overall process of eyestate detection.

FIG. 11 illustrates an example of how a safe cone viewing area can bedefined.

FIG. 12 is a flowchart showing an example of a process of determiningthe driver's attention level based on head movement.

FIG. 13 is a flowchart showing an example of the process of determiningthe driver's attention level based on eye movement (extent of eyeclosure).

FIG. 14 is a flowchart showing an example of the process of detecting arecent prolonged full eye closure event.

FIG. 15 is a flowchart showing an example of the process of detecting arecent prolonged partial eye closure event.

FIG. 16 is a flowchart showing an example of the process of performingeye behavior trend analysis.

FIG. 17 is a flowchart showing an example of the process of calculatingand monitoring driver response/reaction times.

FIG. 18 shows an example of a spatial configuration for driver'sparameter calibration inside a vehicle.

FIG. 19 is a flowchart showing an example of the process of performingdriver's parameter calibration.

FIG. 20 is a flowchart showing an example of the process performed bythe Alerting Subsystem.

DETAILED DESCRIPTION

In this description, references to “an embodiment”, “one embodiment” orthe like, mean that the particular feature, function, structure orcharacteristic being described is included in at least one embodiment ofthe technique introduced here. Occurrences of such phrases in thisspecification do not necessarily all refer to the same embodiment. Onthe other hand, the embodiments referred to also are not necessarilymutually exclusive.

I. Overview

A driver monitoring system that can automatically detect gradualdecrease in driver alertness and warn the driver early enough is animportant step in reducing accidents on the roadway. Introduced here,therefore, is a camera-based driver monitoring system (DMS) thatcontinuously monitors the head and eye movements of the driver andtracks the attention level of the driver. When the attention levelprogressively decreases with time, the DMS introduced here (hereinafteralso called “the system”) can interact with the driver to improve thedriver's attention level. The system continuously captures images of thedriver and applies computer vision and machine learning techniques tocaptured images to extract facial and eye features that can be used totrack changes in the head and eye movement.

Implementing such a camera-based DMS that is cost effective and suitablefor real time driving environment has many challenges. Some of thesignificant challenges are the impact of harsh lighting conditions,vibration due to vehicle movement and natural driver body movement onthe captured images that are used to extract facial and eye features.The DMS introduced here includes a method for eye and face movementtracking that is robust and invariant to lighting, vibration and bodymovement.

The system uses computer vision and machine learning techniques toimprove the safety and convenience of the in-car experience. The systemincludes three primary detection functions: drowsiness detection,distraction detection and gaze detection. These functions are developedby monitoring and tracking changes in the driver facial images. Thesystem continuously tracks head, eyelid and pupil movements. It convertschanges in head, eye and pupil movement to time series events. Machinelearning and computer vision techniques are used to convert imagescaptured by a Near InfraRed (NIR) camera to time series head and eyemovement data for predicting drowsiness, distraction and gaze levels.

Computer vision and machine learning techniques are used to determinethe location of landmarks in the face and eye region. The locations ofthe landmarks in the eye region are used to measure the openness levelof the eye. This measured openness level is then converted to fullyopen, partially open, or closed eye state per image. The eye state istracked continuously to detect measured parameters, such as eye blinkcount, blink rate, eye closure duration, eye open duration, speed ofeyelid closure, and/or speed of eyelid opening. The system continuouslymonitors these parameters for any significant change from establishedconstant threshold values or driver specific calibrated (profile)values.

Environmental changes, such as changing lighting condition, vehiclevibration, and body movement, can introduce errors in the detectedlocation of the landmarks. These errors can significantly impact theaccuracy of the measured openness level of the eye. The proposed systemdescribes a method that can produce robust and invariant measurement ofeye openness level. The proposed approach is robust to variation indistance of the driver's head from camera; yaw, roll and pitchorientation of the head; and human variations such as race, ethnicity,gender, and age. The system can use deep learning and computer visiontechniques to measure the eye openness level to improve robustnessagainst lighting and vibration changes, road environment, and human headmovements.

II. Example Implementation

The system measures the alertness level of the driver by evaluating asequence of images captured by a camera and measuring the openness levelof the eye, and head orientation. The openness level of the eye isconverted to eye state. Eye state in turn is used to detect a collectionof parameters such as blink count, blink duration, eye open duration,eye close duration, speed of eye opening, and speed of eye closure.These measured parameters along with head orientation (yaw, pitch androll) are used to determine the alertness level of the driver.

FIG. 1 illustrates at a high level the hardware configuration of thesystem according to at least some embodiments. As shown, the system 1includes an image capture and store subsystem (also called camerasubsystem) 2, a processing subsystem 3 and an input/output (I/O)subsystem 4. The camera subsystem 2 includes at least one camera (e.g.,a near-IR camera), to capture images of the driver's head, face andeyes. The I/O subsystem 4 includes one or more devices to provide a userinterface 5 capable of receiving inputs such as commands and preferencesfrom the driver (e.g., touchscreen, microphone with voice recognitionsoftware and hardware) and capable of outputting information such asalerts and operating instructions to the driver (e.g., speaker, display,seat vibrator). The processing subsystem 3 is responsible for all of themajor processing functions described herein. The processing subsystem 3can be implemented to include programmable circuitryprogrammed/configured by software and/or firmware, or entirely byspecial-purpose circuitry, or by a combination of such forms. Suchspecial-purpose circuitry (if any) can be in the form of, for example,one or more application-specific integrated circuits (ASICs),programmable logic devices (PLDs), field-programmable gate arrays(FPGAs), system-on-a-chip systems (SOCs), etc. The processing subsystem3 further includes facilities to store digital data, such as one or morememories (which may be read-write, read-only, or a combination thereof),hard drives, flash drives, or the like.

Software or firmware to implement the various functions and stepsdescribed herein may be stored on a machine-readable storage medium andmay be executed by one or more general-purpose or special-purposeprogrammable microprocessors. A “machine-readable medium”, as the termis used herein, includes any mechanism that can store information in aform accessible by a machine (a machine may be, for example, a computer,network device, cellular phone, personal digital assistant (PDA),manufacturing tool, any device with one or more processors, etc.). Forexample, a machine-accessible medium includes recordable/non-recordablemedia, e.g., read-only memory (ROM); random access memory (RAM);magnetic disk storage media; optical storage media; flash memorydevices; etc.

At a functional level, in at least some embodiments, and as shown inFIG. 2 , the system 1 includes three major subsystems: the image captureand store subsystem (or camera subsystem) 2, a driver behavior trackingsubsystem 6 and an alerting subsystem 7, The system 1 further includesfour ancillary functional modules, namely: a system startup module 8, adetect head orientation and feature landmarks module 9, an analyzedriver behavior data module 10 and a dynamic threshold update module 11.Each of these subsystems and modules is described in detail below. Thesystem further creates and maintains several datasets, including astored images dataset 12, a landmarks dataset 13, an eye state dataset14 and a blinks dataset 15.

Each of the elements shown in FIGS. 1 and 2 can be implemented entirelywithin a given vehicle (e.g., an automobile), although in someembodiments, at least portions of one or more elements (e.g., storageand/or processing resources) may be implemented remotely from thevehicle.

The system startup module 8 is responsible for initializing the system1. As part of the initialization sequence, it requests the dynamicthreshold update module 11 to perform calibration. Once the dynamicthreshold update module 11 completes the calibration process, the systemstartup module 8 initializes the other modules.

The camera subsystem 2 continuously captures images of the driver at arate of, for example, approximately 30 frames per second, and stores theimages for further processing. The images can be stored in a localstorage device (not shown), such as a memory or hard drive, or remotelyin the cloud.

A. Head Orientation and Feature Landmarks Detection Module

FIG. 3 shows an example of the overall process performed by the HeadOrientation and Feature Landmarks Detection Module 9. After getting thenext frame at step 301, the process at step 302 analyzes the capturedimage and applies computer vision techniques for image enhancement tominimize the impact of environmental challenges such as vibration andlighting. Some image correction techniques applied include contrastlimited adaptive histogram equalization (CLAHE) and Gaussian blurring.Once the image quality is improved (step 303), the process detects thehead region of interest (ROI) of the driver's head at step 305, and thendetects the driver's face at step 306. If the driver's face is notdetected, a record is inserted into the stored images dataset 12 at step304, indicating frame number and a no-face-detected flag. If a face wasdetected, the top and bottom coordinates of the bounding box containingthe face are predicted by the module.

At step 307 the process uses the previous image's eye region boundingbox coordinates to extract the eye region of the current image andperforms an image similarity check for both left and right eyes. If theextracted left and right eye image comparisons are within the measure ofthreshold for similarity (step 308), then the previous image's landmarkdata are used to add an entry to the landmarks dataset 13 at step 315;otherwise the process continues to step 309.

At step 309 the process uses deep learning techniques to detect theroll, yaw, and pitch of the driver's head from a 2D gray scale image. Atstep 310 the process uses deep learning techniques to detect a fixednumber of landmarks on the face. FIG. 4 shows an example of faciallandmarks on the face of a person such as may be detected at step 310,where each dot represents a landmark.

Step 311 uses the eye corner landmarks from the facial landmarks toextract the eye ROI for both the left and right eye. Step 312 uses deeplearning techniques to detect a fixed number of landmarks around theeyelid contours and the center of the pupil for both left and righteyes, as shown in FIG. 5 . Step 313 treats each predicted coordinate asa time-series datum and applies one or more smoothing techniques such asexponential smoothing, moving average, convolutional smoothing, Kalmansmoothing, and polynomial smoothing. The full data are then written tothe landmarks dataset step 314. After step 314, the process loops backto step 301.

B. Driver Behavior Data Analysis Module

Referring again to FIG. 2 , the Driver Behavior Data Analysis Module 10processes the data stored in the landmarks dataset to determine thedriver's behavior. An example of the process performed by this module isshown in FIG. 6 . Initially, the process gets the next landmark datafrom the landmarks dataset at step 601. The primary purpose of steps 602and 603 is to ensure the driver is consistently maintaining a good viewof the surrounding road environment necessary for safe driving by takinginto consideration driver characteristics, driver behavior, and roadenvironment. Hence, at step 602 the process analyzes the head movementof the driver. Lack of head movement over a prolonged period of time canbe considered an indicator of gradual onset of decreased alertnesslevel. Prolonged head orientation that is outside a “safe viewing cone”area can indicate distraction, such as due to devices, road activities,or interaction with vehicle occupants.

At step 603 the process analyzes the eyelid movement of each eye of thedriver and calculates the eye openness measure (EOM) for each eye. Inparticular, step 603 calculates the robust eye openness measurementangle based on predicted eyelid landmarks. The alert records generatedby this step and step 602 are among several types of alert recordsgenerated by the system. These alert records may be stored in the systemfor further processing and/or they may be used to directly alert thedriver. Step 604 converts the measured EOM value to one of three eyestates namely open, partially open, and closed, as discussed below ingreater detail in connection with FIG. 10 . Step 605 detects any blinksrepresented in the data. Step 606 determines the driver attention levelbased on the eye movement data. In particular, step 606 tracks thefollowing scenarios and generates an alert record for the alert moduleto process:

-   -   Continuous tracking of eye closure duration.    -   Prolonged partial eye closure with minimal change in head        orientation as an indicator of drowsiness onset due to lack of        head movement and partially open eye.    -   Change in driver's eyelid movement pattern over a prolonged        period of time. For example, decrease in blink rate, increase in        eye closure time, or decrease in velocity of eye closure can        indicate onset of driver fatigue or drowsiness.    -   After step 606, the process loops back to step 601.

C. Analysis of Driver Attention Level Based on Head Movement

Step 602 in FIG. 6 , analyzing the driver's attention level based onhead movement, will now be described in greater detail. FIG. 12 shows anexample of the process of determining the driver's attention level basedon head movement. At step 1201 a new data record is retrieved from thelandmarks dataset. The dynamic profile and threshold criteria parametersthat are used in evaluating whether to create an alert record or not arecalculated at step 1202. The alert tracking parameters such as headpitch and yaw are set to initial state at step 1212 if the vehicle isstationary (step 1203), and the process then loops back to step 1201.The head orientation and feature landmarks detection module has aprioritized alert event checking mechanism. The system alerts the driverif his/her head orientation is outside the safe cone viewing area (asshown in FIG. 11 ).

When the vehicle is moving, prolonged head orientation that can indicateonset of decrease in driver attention is tracked. If the measured yaw orpitch angle is greater than the dynamic threshold angles at step 1204,then the corresponding alert frame counter is incremented at step 1205.At step 1206, if the yaw alert counter exceeds the dynamic yaw alertthreshold counter or the pitch alert counter exceeds the dynamic pitchalert threshold counter, then an alert record is generated and therecord is inserted into the alerts dataset at step 1211.

If neither threshold is exceeded at step 1206, then at step 1207 a batchof head orientation data of size k is retrieved, where parameter k is adynamic value retrieved from the profile and threshold data module. Ifthe standard deviation, calculated for the batch, of the yaw or pitchangles indicates very low head movement (by being less than acorresponding threshold), this implies that the driver is not routinelyscanning the road environment. Therefore, if the calculated standarddeviation value is less than the dynamically calculated standarddeviation threshold values from the profile and threshold data module atstep 1208, then an alert record is generated and inserted into alertsdataset at step 1211.

If no alert is detected, then a check is made to see if a previous headorientation-based alert was created. If no clear alert record exists forthat alert record, then a clear alert record is generated and insertedinto the alerts dataset at step 1210 to reset the alert records.

D. Calculation of EOM

A method for calculating the EOM, per step 603 in FIG. 6 , is nowdiscussed in greater detail in relation to FIGS. 7A, 7B and 9 . The eyelandmark record from the landmarks dataset is retrieved at step 901(FIG. 9 ), and the upper and lower contours of the eye are plottedprogrammatically at step 902. At step 903 the left and rightintersection points of the upper eyelid contour 71 and lower eyelidcontour 72 are identified as landmarks P₀ and P₁, respectively, as shownin FIGS. 7A, 7B and 8 , and a line segment P₀P₁ that connects P₀ to P₁is then plotted. The line segment P₀P₁ is then segmented into four equalsegments at step 904, such that it produces points P_(A), P_(B) andP_(C), whereP ₀ P _(A) =P _(A) P _(B) =P _(B) P _(C) =P _(C) P ₁

Line segments perpendicular to P₀P₁ and at P_(A), P_(B) and P_(C)intersecting upper and lower eyelid contours are then plotted at step905. The intersection points of the perpendicular lines at P_(A), P_(B)and P_(C) with the upper and lower eyelid contours, designated as P₂,P₄, P₆, P₃, P₅ and P₇ in FIG. 7A, are then identified at steps 906 and907, respectively.

The process then defines a plurality of triangles, such that thevertices of each triangle are formed by a different 3-tuple of landmarkson the upper and lower eyelid contours, and where two of those landmarksin the 3-tuple are located on a different eyelid contour (upper orlower) from the third landmark in the 3-tuple. Using the properties of atriangle, therefore, the angles at each of eyelid landmarks P₂, P₄, P₆,P₃, P₅ and P₇ (i.e., excluding the endpoints P₀ and P₁) are calculatedat step 908 in accordance with equations (1) through (3) below, and theaverage of the angles is calculated at step 909, per equation (4) below.Consider for example the triangle formed by the eyelid landmarks P₄, P₃and P₇ as shown in FIG. 7B:a distance(P ₃ ,P ₇)b distance(P ₄ ,P ₃)c distance(P ₄ ,P ₇)a ² =b ² +c ²−2·b·c·cos α(P ₄)  (1)

$\begin{matrix}{{\cos{\alpha\left( P_{4} \right)}} = \frac{\left( {b^{2} + c^{2} - a^{2}} \right)}{2 \cdot b \cdot c}} & (2)\end{matrix}$

$\begin{matrix}{{\alpha\left( P_{4} \right)} = {\arccos\left( \frac{\left( {b^{2} + c^{2} - a^{2}} \right)}{2 \cdot b \cdot c} \right)}} & (3)\end{matrix}$

$\begin{matrix}{{EOM} = \frac{\left( {{\sum}_{k = 1}^{n}\alpha} \right)}{n}} & (4)\end{matrix}$Where

-   -   α(P₄), is the angle at landmark P₄,    -   EOM, is the Eye Openness Measurement,    -   n, is the number of angles calculated, and    -   α, is the angle at each eyelid landmark with respect to other        eyelid landmarks.

An alternate method used for measuring eye openness level is illustratedschematically in FIG. 8 . This alternate method involves measuring theheight of the eyelid at selected points and averaging the eye height bythe number of such measurements multiplied by the width of the eye inaccordance with equation (5).

$\begin{matrix}{{EHWR} = \frac{{{P_{2} - P_{3}}} + {{P_{4} - P_{5}}} + {{P_{6} - P_{7}}}}{3{{P_{1} - P_{0}}}}} & (5)\end{matrix}$

In a real-time environment, the eyelid landmarks detected by the systemcan vary from frame to frame. These variations have a negative impact onthe accuracy of eyelid movement tracking. The accuracy of the detectionof landmarks at the edge of the eyes is more prone to variation than thelandmarks on the eyelid. These variations on successive images can havesignificant negative impact on the accuracy of the eyelid tracking.Therefore, the EOM calculation method of FIGS. 7A, 7B and 9 attempts toreduce the impact of eye edge landmarks by avoiding them when measuringangles among the eyelid landmarks. The alternate method of FIG. 8 ismore susceptible to variation in the eye edge landmarks because theratio is formed by dividing the height of the eyelid by the width of theeyelid (i.e., the distance between the edges of the eye).

Since the width of the eye is almost twice the height of the eye, theratio produced by this alternate EOM method is often less than 0.5 whenthe eye is fully open and tends towards zero as the eyelid closes. As aresult, the inaccuracies in landmark detection on eye angle ratio (EAR)tend to have significantly more negative impact as the eyelid closes.The EOM calculation method of FIGS. 7A, 7B and 9 on the other hand tendsto have larger range of angles from 30° to 180° for fully open eyelid toclosed eyelid. The angle between the landmarks on the eyelid increasesas the eyelid closes. This behavior of the EOM calculation method ofFIGS. 7A, 7B and 9 reduces the impact of inaccuracies in eyelid landmarkdetection in comparison to at least the method of FIG. 8 .

E. Calculation of Eye State

Step 604 in FIG. 6 , eye state calculation, will now be described ingreater detail. FIG. 10 illustrates an example of the overall processthat may be performed in step 604. The process begins by retrieving thenext EOM value from the EOM dataset (step 1001), and based onpre-configured thresholds (step 1002 and step 1003) representingpercentage of eye closure, categorizing the EOM value into open eye(step 1009), partially closed eye (step 1010) or closed eye state (step1011), which is then inserted into the eye state dataset 14 at step1004. Partially closed eye state (step 1010) is used to determine thedriver alertness level when the driver is looking down such that the eyeis not fully closed, but the percentage of open eye is not sufficientfor safe driving. That is, with a partially closed eye, the driver doesnot meet the criteria for “safe cone viewing area”. The “safe coneviewing area” is the area of the windshield between the driver door andthe rear-view mirror, as shown in FIG. 11 . Once the eye state isdetermined, the time series eye state data set is smoothed at step 1005using, for example, any of the smoothing methods described above.

Next, the process checks for eye blinks at step 1006. If a blink isdetected, blink related parameters such as blink duration, eye closureduration, eyelid closure speed, and/or eyelid open speed are calculated,and a blink record is created and stored in the blinks dataset 15. Theblinks dataset 15 includes two subsets, called the blinks-per-seconddataset and the blinks-per-minute dataset (not shown). If the eyeclosure duration exceeds the microsleep duration threshold, a flag isset to indicate a microsleep event in the blink record.

Next the process checks whether one second of blink data has beencollected at step 1007. If so, a set of per-second parameters arecalculated, a per-second record is created, and the record is stored ina blinks-per-second dataset at step 1007. The parameters can include,for example, blink rate per second, average blink duration, average eyeclosure, average speed of eyelid closure, average speed of eyelidopening, and/or count of eye closures that exceed the microsleepduration threshold. Similarly, a similar set of parameters arecalculated for each minute of record and stored in a blinks-per-minutedataset at step 1008.

F. Analysis of Driver Attention Level Based on Eye Movement:

FIG. 13 shows in greater detail, an example of step 606 (FIG. 6 ),determining the driver's attention level based on eye movement. Thisprocess is responsible for detecting unsafe driving based on acollection of analyses applied on eye movement datasets. As in theprocess of FIG. 12 , when an unsafe event is detected, an alert recordrepresenting the type of unsafe event is created and inserted into thealerts dataset. Similarly, when the unsafe event clears, a correspondingclear alert record is also inserted to the alerts dataset.

Step 1301 gets the eye state record, calculated in step 606 of FIG. 6 .Step 1302 continuously analyzes the eye state data to detect any eyeclosure duration that is higher than the dynamic eye closed statethreshold value. Step 1303 detects continuous partially closed eye statein conjunction with head orientation. It is possible to have partiallyclosed eye states due to reclined seat and head orientation. Thesescenarios should not create an alarm state. Therefore, step 1303 takesinto consideration the head orientation, duration, and eye state foranalysis. Step 1304 detects changes in eye closure behavior over timethat indicate onset of decrease in driver attention level. This stepanalyzes parameters such as blink rate, speed of eyelid movement, andduration of eye closure.

G. Detection of Recent Prolonged Full Eye Closure Event

FIG. 14 shows in greater detail, an example of the process included instep 1303, detecting a recent prolonged full eye closure event. Thisprocess at step 1401 initially retrieves driver profile and thresholddata (e.g., dynamic threshold parameters k, m and n, discussed below)from the dynamic threshold update module 11 (FIG. 2 ). At steps 1402 and1403, the most recent batch of eye states is analyzed and the number ofcontinuous frames for which the eye state is closed is counted. For eachframe where the eye state is closed, the closed eye frame count isincremented (step 1404). The closed eye frame count is compared at step1405 against the dynamic eye state closed count threshold (retrievedfrom profile and threshold data at step 1401), and if the closed eyeframe count is greater, a recent prolonged eye closure alert record iscreated at step 1406 and inserted into the alerts dataset at step 1407;otherwise, the process loops back to step 1401. If during the process oflooking for continuous eye closed state at step 1403, a partial closedor open eye state is encountered, the closed eye frame count is reset atstep 1408 and checked for a previous recent prolonged eye closure alertrecord without a clear alert record at step 1409. If it exists, a clearrecent prolonged eye closure alert record is created and inserted intothe alerts dataset at step 1410; otherwise, the process loops back tostep 1401.

H. Detection of Recent Prolonged Partial Eye Closure Event

FIG. 15 shows in greater detail, an example of the process included instep 1303, detecting a recent prolonged partial eye closure event. Thisstep counts the number of continuous partially closed eye state frames.Continuous partial eye closure state may also happen due to a reclineddriver seat. When the seat is reclined, a driver tends to adjust theirhead such that the head remains within the safe driving cone. Such anorientation, by itself, is not a concern for safety as long as thedriver maintains safe driving behavior. Therefore, it is important totake into consideration not only the duration of partial eye closurestate but also the head orientation and corresponding duration of thehead orientation.

This process (FIG. 15 ) at step 1501 initially retrieves driver profileand threshold data for partial eye closed state (e.g., dynamic thresholdparameters k, m and n, discussed below) from the dynamic thresholdupdate module 11 (FIG. 2 ). The most recent batch of eye states isanalyzed and the number of continuous frames for which the eye state ispartially closed is counted at steps 1502 and 1503. For each frame wherethe eye state is partially closed, the partially closed eye frame countis incremented (step 1504). If at step 1505 the partially closed eyeframe count is greater than the dynamic eye state partially closed countthreshold retrieved from profile and threshold data at step 1501, thenat step 1506 the process calculates the standard deviation of head pitchover n multiples of partial eye closure frame count, where parameter nis determined by the profile and threshold module. Otherwise, theprocess loops back to step 1501. After step 1506, if the standarddeviation is less than the corresponding threshold for head pitchstandard deviation (indicating very little or narrow head movement) atstep 1507, then an alert record indicating prolonged partial eye closureevent is created at step 1508 and is inserted into alerts dataset atstep 1509. Otherwise, the process loops back to step 1501. If a partialeye state is not detected at step 1503, then the partially closed eyeframe count is reset at step 1510, and if a previous alert exists with aclear alert message (step 1511), then a clear alert message forprolonged partial eye closure event is created and inserted into thealerts dataset at step 1512.

I. Eye Behavior Trend Analysis

FIG. 16 shows in greater detail, an example of the process included instep 1304 (FIG. 13 ), eye behavior trend analysis. This analysisinvolves projecting future behavior based on historical behavioral data.The module analyzes the trend in the eye closure duration stored inblinks-per-minute dataset. The process initially retrieves driverprofile and threshold data (e.g., dynamic threshold parameters k, m andn, discussed below) from the dynamic threshold update module 11 (FIG. 2) at step 1601, and retrieves data from the blinks-per-minute dataset atstep 1602. If the eye closure duration shows a trend of k % increase fora sample of m minutes at step 1603, that is deemed indicative of theonset of drowsiness. In that event, an alert of type ‘change in eyeclosure trend’ is created and the record is inserted into alerts datasetat step 1604. The parameter k is a specified threshold percentage value,and parameter m is a specified number of minutes. Similarly, if themicrosleep count does not decrease by at least one count in a sample ofn minutes at step 1605, that is deemed indicative of an ongoingdrowsiness state. In that event, therefore, an alert of type ‘change inmicrosleep trend’ is created and the record is inserted into alertsdataset at step 1606. If no alert condition is detected at step 1603 orstep 1605, then a check is made for a previous alert of type ‘change ineye closure trend’ or ‘change in microsleep trend’ at step 1607 or 1609,respectively. Otherwise the process loops back to step 1601. If aprevious alert exists without a corresponding clear alert request, thena record is created to clear the ‘change in eye closure trend’ and isinserted into alerts dataset. After step 314, the process loops back tostep 301.

J. Dynamic Threshold Update Module

The dynamic threshold update module 11 (FIG. 2 ) is responsible forproviding the thresholds used for eye state and head movement comparisonto determine driver alertness level. The thresholds can be broadlyclassified into time-based threshold values and angle-based thresholdvalues. Time-based threshold values are used in deciding how long thedriver can remain inattentive safely before such inattentivenessrepresents a decrease in alertness. Time-based threshold values are alsoof two types. One type deals with immediate events and the other dealswith trends over a longer period of time. Angle-based threshold valuescan be broadly categorized into two types, one based on head orientation(roll, yaw, and pitch) and the other based on eye movement (eyelidopenness measure).

The time-based threshold values deal with reaction or response timerequired for a safe level of driver attention. The required reactiontime can vary depending on the influence of various environmentalparameters. As the driving environment changes, the comparisonthresholds are updated proportionately to keep up with the changingenvironment. Parameters that may influence the driver's reaction orresponse time to an event while driving can be broadly categorized intothe following types:

-   -   Driver parameters, for example: age, physical limitations, EOM,        blink rate, eye closure time, head orientation angles    -   Vehicle parameters, for example: length, weight, speed, route        maps    -   Road parameters, for example: traffic congestion, road type        (straight, curvy)    -   Weather parameters, for example: time of day, season

These parameters tend to have significant impact on a driver's reactiontime to a driving event. Therefore, the dynamic threshold update modulecan take into consideration some or all of these parameters incalculating the threshold parameters. For example, consider thefollowing scenario: A 35-year-old male is driving a sedan at 100 km/houron a summer Sunday morning on a highway with low traffic. Typicalresponse time for a young adult is around 200 ms. At 100 km/hour thesedan travels 27 meters per second. A typical sedan length is 4.5meters. Therefore in 200 ms, the vehicle would have travelled 5.4 meterswhich is more than the length of a sedan car. Therefore, it is importantto take into consideration the above-mentioned input parameters whendeciding how long a driver can remain inattentive before it becomesunsafe. The reaction time also increases due to the influence ofparameters such as age, drowsiness/fatigue, and distraction.

The dynamic threshold update module input data sources include:

-   -   Vehicle parameters: Vehicle status information including but not        limited to speed, turn signal, and distance to objects in front        of the vehicle, and camera frame rate.    -   Driver parameters: Driver parameters stored locally on a device,        driver mobile phone, and/or cloud services.    -   Road parameters: Road parameters can be obtained from navigation        systems, route maps, road infrastructure, cloud services, and/or        driver mobile phone.    -   Weather parameters: Weather parameters can be obtained from road        infrastructure, driver mobile phone, and/or cloud services.

Each input parameter has a preconfigured safe response time. Forexample, Tables 1, 2 and 3 describe examples of the input parameters andcorresponding safe response times and weights.

TABLE 1 Speed Influence Low Medium High fractional Speed (km/h) (0 to30) (31 to 50) (51 to 100) weight Response (ms) S₁ S₂ S₃ ω_(S)

TABLE 2 Driver Age Influence fractional Age (years) 20 to 40 41 to 50 51to 100 weight Response (ms) A₁ A₂ A₃ ω_(A)

TABLE 3 Traffic Congestion Influence Traffic fractional Congestion LowMedium High weight Response (ms) T₁ T₂ T₃ ω_(T)

Each reaction time threshold parameter can be a weighted sum of theseinput parameters. The general equation for a time-based thresholdparameter is:

${TRT} = \frac{\left( {{\sum}_{k = 1}^{n}\omega_{k}*{response\_ time}_{k}} \right)}{n}$where

-   -   TRT is threshold response time    -   ω_(k) is weight contribution of the parameter    -   response_time_(k) is the independent response_time of the        parameter    -   n is the number of input parameters

Angle threshold values for head orientation can be based on generic datasupplied with the system or calibrated roll, yaw, and pitch values forthe driver. These values can be measured from calibration. Theyrepresent a head orientation suitable for the safe cone viewing area.The eye openness level measure-based thresholds can also be obtainedeither from custom driver calibration or from generic data supplied withthe system.

An example of a process for calculating and monitoring the driver'sresponse/reaction times is shown in FIG. 17 . After the driver beginsdriving the car (step 1701), the system at step 1702 gets inputparameters from the driver's profile as well as vehicle, road andweather conditions. The system then calculates response/reaction timesand updates the thresholds continuously at step 1703. At step 1704 thesystem determines whether or not the driver is driving within the “safecone viewing area” and whether a recent prolonged partial eye closureevent has been detected. If either condition is detected, the systemproceeds to step 1705, in which the system updates the count of unsafedriving time. If neither condition is detected, the process loops backto step 1703. If the unsafe driving time exceeds a correspondingthreshold at step 1706, then the system creates an alert record andinserts it into the alerts dataset at step 1707.

K. Driver Parameter Calibration

The Dynamic Threshold Update Module 11 is also responsible for obtainingthe driver parameters, such as age, physical limitations, headorientation angles, and eye openness level measures through calibration.An example of a spatial configuration for driver's parameter calibrationinside a vehicle is shown in FIG. 18 , and an example of the process ofdriver's parameter calibration is shown in FIG. 19 .

To perform calibration (e.g., when the car is first started), theprocess initially performs a facial recognition to identify the driverat step 1901. If the driver is identified (step 1902), then the driverprofile data are retrieved from local storage at step 1903. A check isperformed for a newer version of the driver profile by requesting anupdate from the driver's mobile device and/or a cloud service. If anupdate is available then it is stored locally.

If the driver is not identified, then at step 1904 the process performsa calibration to create a profile for the driver. At step 1905 theprocess gets the driver's age and physical limitations (if any). Theprocess then at step 1906 provides audio and visual instructions to thedriver to look at specific locations inside the vehicle, examples ofwhich are indicated by the diamond shapes in FIG. 18 . At each location,the module at step 1907 captures a batch of images for a period of, forexample, five seconds. The captured images are then used to determinehead orientation angles and eye openness level measures using facial andeye landmarks at step 1908. The measured values are then stored at step1909 in local storage system, cloud service, and/or the driver's mobiledevice for future access.

L. Alerting Subsystem

The Alerting Subsystem 7 (FIG. 2 ) generates the alerts. An example ofthe overall process performed by this subsystem is shown in FIG. 20 .All alerts recorded in the alerts dataset are processed (step 2001) bythe Alerting Subsystem 7. The process in FIG. 20 checks whether thealert level recorded in the dataset is high at step 2002. If the alertlevel is high, the process informs the driver at step 2003 by any one ormore methods, such as seat/steering wheel vibration, visual display,and/or audible alert. If the alert level is not high, the process stopsall previously generated alerts at step 2004. After step 2003, theprocess loops back to step 2001.

III. Examples of Certain Embodiments

Certain embodiments of the technology introduced herein are summarizedin the following numbered examples:

-   -   1. A method comprising:    -   obtaining image data representing images of an eye of a driver,        the images having been captured while the driver is operating a        vehicle;    -   computing an eye openness measure of the eye of the driver,        based on the image data;    -   determining that the driver is in an unsafe state for driving        based on the eye openness measure; and    -   generating an alert, while the driver is operating the vehicle,        based on determining that the driver is in the unsafe state.    -   2. A method as recited in example 1, wherein computing an eye        openness measure of the eye of the driver comprises determining        an eye state for the eye, and wherein the eye state is from the        set of eye states consisting of: a fully open state, a partially        open state, and a closed state.    -   3. A method as recited in example 1 or example 2, further        comprising:    -   determining a parameter set based on the eye state, the        parameter set including at least one of a blink count, blink        rate, eye closure duration, eye open duration, speed of eyelid        closure, or speed of eyelid opening; and    -   applying at least one parameter of the parameter set to a        threshold or profile to identify an amount of deviation.    -   4. A method as recited in any of examples 1 through 3, further        comprising:    -   determining a plurality of thresholds;    -   wherein said determining that the driver is in an unsafe state        for driving comprises applying acquired or computed data to the        plurality of thresholds, and wherein the plurality of thresholds        includes a plurality of time-based thresholds and a plurality of        angle-based thresholds.    -   5. A method as recited in any of examples 1 through 4, wherein        at least one of the time-based thresholds relates to driver        reaction time.    -   6. A method as recited in any of examples 1 through 5, wherein        at least a first one of the angle-based thresholds relates to        the eye openness measure.    -   7. A method as recited in any of examples 1 through 6, wherein        at least a second one of the angle-based thresholds relates to a        head orientation of the driver.    -   8. A method as recited in any of examples 1 through 7, wherein:    -   at least one of the time-based thresholds relates to driver        reaction time;    -   at least a first one of the angle-based thresholds relates to        the eye openness measure; and    -   at least a second one of the angle-based thresholds relates to a        head orientation of the driver.    -   9. A method as recited in any of examples 1 through 8, further        comprising dynamically updating the plurality of thresholds        while the driver is operating the vehicle.    -   10. A method as recited in any of examples 1 through 9, wherein        computing the eye openness measure of the eye of the driver        comprises:    -   generating an upper eyelid contour of the eye and a lower eyelid        contour of the eye based on the image data representing images        of the eye;    -   identifying a plurality of landmarks on each of the upper eyelid        contour and the lower eyelid contour;    -   defining a plurality of triangles, such that each of the        plurality of triangles has vertices formed by a different        3-tuple of landmarks among the plurality of landmarks;    -   computing interior angles of the plurality of triangles; and    -   computing the eye openness measure of the eye as a function of        the interior angles.    -   11. A method as recited in any of examples 1 through 10, wherein        two of the three vertices of each triangle are located on a        different one of the upper eyelid contour and the lower eyelid        contour than the third vertex of the triangle.    -   12. A method as recited in any of examples 1 through 9, wherein        computing an eye openness measure of the eye of the driver        comprises:    -   obtaining landmark data of the eye;    -   generating an upper eyelid contour of the eye and a lower eyelid        contour of the eye based on the landmark data;    -   plotting a first line between a first intersection point of the        upper eyelid contour and the lower eyelid contour and a second        intersection point of the upper eyelid contour and the lower        eyelid contour;    -   dividing the first line into at least four equal segments;    -   plotting a plurality of second lines, wherein each of the        plurality of second lines passes through an intersection point        between a different two of the at least four equal segments of        the first line, perpendicularly to the first line;    -   identifying a plurality of landmarks on each of the upper eyelid        contour and the lower eyelid contour, the plurality of landmarks        comprising each intersection point between a line of the        plurality of second lines and either the upper eyelid contour or        the lower eyelid contour;    -   defining a plurality of triangles, such that each of the        plurality of triangles has vertices formed by a different        3-tuple of landmarks among the plurality of landmarks, wherein        two of the landmarks in each said 3-tuple are located on a        different one of the upper eyelid contour and the lower eyelid        contour than the third landmark in the 3-tuple;    -   computing each interior angle of each of the plurality of        triangles; and    -   computing the eye openness measure of the eye as a function of        the average of all of the interior angles of the plurality of        angles.    -   13. A method as recited in any of examples 1 through 12, further        comprising:    -   detecting a prolonged partial eye closure based on a head        orientation angle and an eye state of the eye.    -   14. A method as recited in any of examples 1 through 13, further        comprising:    -   calculating an eye behavior trend for the driver based on        detected eye closure and microsleep patterns of the driver.    -   15. A non-transitory machine-readable storage medium having        instructions stored thereon, execution of which by a processing        system in a vehicle causes the processing system to perform a        process comprising:    -   obtaining image data representing images of an eye of a driver        of the vehicle, the images having been captured while the driver        is operating the vehicle;    -   computing an eye openness measure of the eye of the driver,        based on the image data;    -   determining that the driver is in an unsafe state for driving        based on the eye openness measure; and    -   generating an alert, while the driver is operating the vehicle,        based on determining that the driver is in the unsafe state.    -   16. A non-transitory machine-readable storage medium as recited        in example 15, wherein computing an eye openness measure of the        eye of the driver comprises determining an eye state for the        eye, and wherein the eye state is from the set of eye states        consisting of: a fully open state, a partially open state, and a        closed state.    -   17. A non-transitory machine-readable storage medium as recited        in example 15 or example 16, said process further comprising:    -   determining a parameter set based on the eye state, the        parameter set including at least one of a blink count, blink        rate, eye closure duration, eye open duration, speed of eyelid        closure, or speed of eyelid opening; and    -   applying at least one parameter of the parameter set to a        threshold or profile to identify an amount of deviation.    -   18. A non-transitory machine-readable storage medium as recited        in any of examples 15 through 17, said process further        comprising:    -   determining a plurality of thresholds;    -   wherein said determining that the driver is in an unsafe state        for driving comprises applying acquired or computed data to the        plurality of thresholds, and wherein the plurality of thresholds        includes a plurality of time-based thresholds and a plurality of        angle-based thresholds.    -   19. A non-transitory machine-readable storage medium as recited        in any of examples 15 through 18, wherein:    -   at least one of the time-based thresholds relates to driver        reaction time;    -   at least a first one of the angle-based thresholds relates to        the eye openness measure; and    -   at least a second one of the angle-based thresholds relates to a        head orientation of the driver.    -   20. A non-transitory machine-readable storage medium as recited        in any of examples 15 through 19, wherein computing the eye        openness measure of the eye of the driver comprises:    -   generating an upper eyelid contour of the eye and a lower eyelid        contour of the eye based on the image data representing images        of the eye;    -   identifying a plurality of landmarks on each of the upper eyelid        contour and the lower eyelid contour;    -   defining a plurality of triangles, such that each of the        plurality of triangles has vertices formed by a different        3-tuple of landmarks among the plurality of landmarks;    -   computing interior angles of the plurality of triangles; and    -   computing the eye openness measure of the eye as a function of        the interior angles.    -   21. A system for monitoring driver alertness in a vehicle, the        system comprising:    -   a camera to capture image data representing images of an eye of        a driver of the vehicle while the driver is operating the        vehicle;    -   an output device; and    -   a processing subsystem, coupled to the camera and the output        device, to receive the image data from the camera;        -   compute an eye openness measure of the eye of the driver,            based on the image data;        -   determine that the driver is in an unsafe state for driving            based on the eye openness measure;        -   generate an alert, while the driver is operating the            vehicle, based on determining that the driver is in the            unsafe state; and        -   cause the output device to output to the driver a signal            indicative of the alert.    -   22. A system as recited in example 21, wherein to compute an eye        openness measure of the eye of the driver comprises to determine        an eye state for the eye, and wherein the eye state is from the        set of eye states consisting of: a fully open state, a partially        open state and a closed state.    -   23. A system as recited in example 21 or example 22, wherein the        processing subsystem further is to:    -   determine a parameter set based on the eye state, the parameter        set including at least one of a blink count, blink rate, eye        closure duration, eye open duration, speed of eyelid closure or        speed of eyelid opening; and    -   apply at least one parameter of the parameter set to a threshold        or profile to identify an amount of deviation.    -   24. A system as recited in any of examples 21 through 23,        wherein the processing subsystem further is to:    -   determine a plurality of thresholds;    -   wherein to determine that the driver is in an unsafe state for        driving comprises to apply acquired or computed data to the        plurality of thresholds, and wherein the plurality of thresholds        include a plurality of time-based thresholds and a plurality of        angle-based thresholds.    -   25. A system as recited in any of examples 21 through 24,        wherein:    -   at least one of the time-based thresholds relates to driver        reaction time;    -   at least a first one of the angle-based thresholds relates to        the eye openness measure; and    -   at least a second one of the angle-based thresholds relates to a        head orientation of the driver.    -   26. A system as recited in any of examples 21 through 25,        wherein the processing subsystem further is to update the        plurality of thresholds while the driver is operating the        vehicle.    -   27. A system as recited in any of examples 21 through 26,        wherein to compute the eye openness measure of the eye of the        driver comprises:    -   generating an upper eyelid contour of the eye and a lower eyelid        contour of the eye based on the image data representing images        of the eye;    -   identifying a plurality of landmarks on each of the upper eyelid        contour and the lower eyelid contour;    -   defining a plurality of triangles, such that each of the        plurality of triangles has vertices formed by a different        3-tuple of landmarks among the plurality of landmarks;    -   computing interior angles of the plurality of triangles; and    -   computing the eye openness measure of the eye as a function of        the interior angles.

Unless contrary to physical possibility, it is envisioned that (i) themethods/steps described herein may be performed in any sequence and/orin any combination, and that (ii) the components of respectiveembodiments may be combined in any manner.

The machine-implemented operations described above can be implemented byprogrammable circuitry programmed/configured by software and/orfirmware, or entirely by special-purpose circuitry, or by a combinationof such forms. Such special-purpose circuitry (if any) can be in theform of, for example, one or more application-specific integratedcircuits (ASICs), programmable logic devices (PLDs), field-programmablegate arrays (FPGAs), system-on-a-chip systems (SOCs), etc.

Software or firmware to implement the techniques introduced here may bestored on a machine-readable storage medium and may be executed by oneor more general-purpose or special-purpose programmable microprocessors.A “machine-readable medium”, as the term is used herein, includes anymechanism that can store information in a form accessible by a machine(a machine may be, for example, a computer, network device, cellularphone, personal digital assistant (PDA), manufacturing tool, any devicewith one or more processors, etc.). For example, a machine-accessiblemedium includes recordable/non-recordable media (e.g., read-only memory(ROM); random access memory (RAM); magnetic disk storage media; opticalstorage media; flash memory devices; etc.), etc.

The term “logic”, as used herein, means: a) special-purpose hardwiredcircuitry, such as one or more application-specific integrated circuits(ASICs), programmable logic devices (PLDs), field programmable gatearrays (FPGAs), or other similar device(s); b) programmable circuitryprogrammed with software and/or firmware, such as one or more programmedgeneral-purpose microprocessors, digital signal processors (DSPs) and/ormicrocontrollers, system-on-a-chip systems (SOCs), or other similardevice(s); or c) a combination of the forms mentioned in a) and b).

Any or all of the features and functions described above can be combinedwith each other, except to the extent it may be otherwise stated aboveor to the extent that any such embodiments may be incompatible by virtueof their function or structure, as will be apparent to persons ofordinary skill in the art. Unless contrary to physical possibility, itis envisioned that (i) the methods/steps described herein may beperformed in any sequence and/or in any combination, and that (ii) thecomponents of respective embodiments may be combined in any manner.

Although the subject matter has been described in language specific tostructural features and/or acts, it is to be understood that the subjectmatter defined in the appended claims is not necessarily limited to thespecific features or acts described above. Rather, the specific featuresand acts described above are disclosed as examples of implementing theclaims and other equivalent features and acts are intended to be withinthe scope of the claims.

What is claimed is:
 1. A method comprising: determining a plurality ofthresholds including a plurality of time-based thresholds and aplurality of angle-based thresholds: obtaining image data representingimages of an eye of a driver, the images having been captured while thedriver is operating a vehicle; computing an eye openness measure of theeye of the driver, based on the image data; determining that the driveris in an unsafe state for driving based on the eye openness measure,wherein said determining comprises applying obtained or computed data tothe plurality of thresholds; and generating an alert, while the driveris operating the vehicle, based on determining that the driver is in theunsafe state.
 2. The method as recited in claim 1, wherein computing aneye openness measure of the eye of the driver comprises determining aneye state for the eye, and wherein the eye state is from a set of eyestates consisting of: a fully open state, a partially open state, and aclosed state.
 3. The method as recited in claim 2, further comprising:determining a parameter set based on the eye state, the parameter setincluding at least one of a blink count, blink rate, eye closureduration, eye open duration, speed of eyelid closure, or speed of eyelidopening; and applying at least one parameter of the parameter set to athreshold or profile to identify an amount of deviation.
 4. The methodas recited in claim 1, wherein at least one of the time-based thresholdsrelates to driver reaction time.
 5. The method as recited in claim 1,wherein at least a first one of the angle-based thresholds relates tothe eye openness measure.
 6. The method as recited in claim 1, whereinat least a second one of the angle-based thresholds relates to a headorientation of the driver.
 7. The method as recited in claim 1, wherein:at least one of the time-based thresholds relates to driver reactiontime; at least a first one of the angle-based thresholds relates to theeye openness measure; and at least a second one of the angle-basedthresholds relates to a head orientation of the driver.
 8. The method asrecited in claim 1, further comprising dynamically updating theplurality of thresholds while the driver is operating the vehicle. 9.The method as recited in claim 1, wherein computing the eye opennessmeasure of the eye of the driver comprises: generating an upper eyelidcontour of the eye and a lower eyelid contour of the eye based on theimage data representing images of the eye; identifying a plurality oflandmarks on each of the upper eyelid contour and the lower eyelidcontour; defining a plurality of triangles, such that each of theplurality of triangles has vertices formed by a different 3-tuple oflandmarks among the plurality of landmarks; computing interior angles ofthe plurality of triangles; and computing the eye openness measure ofthe eye as a function of the interior angles.
 10. The method as recitedin claim 6, wherein two of the three vertices of each triangle arelocated on a different one of the upper eyelid contour and the lowereyelid contour than the third vertex of the triangle.
 11. The method asrecited in claim 1, wherein computing an eye openness measure of the eyeof the driver comprises: obtaining landmark data of the eye; generatingan upper eyelid contour of the eye and a lower eyelid contour of the eyebased on the landmark data; plotting a first line between a firstintersection point of the upper eyelid contour and the lower eyelidcontour and a second intersection point of the upper eyelid contour andthe lower eyelid contour; dividing the first line into at least fourequal segments; plotting a plurality of second lines, wherein each ofthe plurality of second lines passes through an intersection pointbetween a different two of the at least four equal segments of the firstline, perpendicularly to the first line; identifying a plurality oflandmarks on each of the upper eyelid contour and the lower eyelidcontour, the plurality of landmarks comprising each intersection pointbetween a line of the plurality of second lines and either the uppereyelid contour or the lower eyelid contour; defining a plurality oftriangles, such that each of the plurality of triangles has verticesformed by a different 3-tuple of landmarks among the plurality oflandmarks, wherein two of the landmarks in each said 3-tuple are locatedon a different one of the upper eyelid contour and the lower eyelidcontour than the third landmark in the 3-tuple; computing each interiorangle of each of the plurality of triangles; and computing the eyeopenness measure of the eye as a function of the average of all of theinterior angles of the plurality of angles.
 12. The method as recited inclaim 1, further comprising: detecting a prolonged partial eye closurebased on a head orientation angle and an eye state of the eye.
 13. Themethod as recited in claim 1, further comprising: calculating an eyebehavior trend for the driver based on detected eye closure andmicrosleep patterns of the driver.
 14. A non-transitory machine-readablestorage medium having instructions stored thereon, execution of which bya processing system in a vehicle causes the processing system to performa process comprising: determining a plurality of thresholds including aplurality of time-based thresholds and a plurality of angle-basedthresholds: obtaining image data representing images of an eye of adriver of the vehicle, the images having been captured while the driveris operating the vehicle; computing an eye openness measure of the eyeof the driver, based on the image data; determining that the driver isin an unsafe state for driving based on the eye openness measure,wherein said determining comprises applying obtained or computed data tothe plurality of thresholds; and generating an alert, while the driveris operating the vehicle, based on determining that the driver is in theunsafe state.
 15. The non-transitory machine-readable storage medium asrecited in claim 14, wherein computing an eye openness measure of theeye of the driver comprises determining an eye state for the eye, andwherein the eye state is from a set of eye states consisting of: a fullyopen state, a partially open state, and a closed state.
 16. Thenon-transitory machine-readable storage medium as recited in claim 15,said process further comprising: determining a parameter set based onthe eye state, the parameter set including at least one of a blinkcount, blink rate, eye closure duration, eye open duration, speed ofeyelid closure, or speed of eyelid opening; and applying at least oneparameter of the parameter set to a threshold or profile to identify anamount of deviation.
 17. The non-transitory machine-readable storagemedium as recited in claim 14, wherein: at least one of the time-basedthresholds relates to driver reaction time; at least a first one of theangle-based thresholds relates to the eye openness measure; and at leasta second one of the angle-based thresholds relates to a head orientationof the driver.
 18. The non-transitory machine-readable storage medium asrecited in claim 14, wherein computing the eye openness measure of theeye of the driver comprises: generating an upper eyelid contour of theeye and a lower eyelid contour of the eye based on the image datarepresenting images of the eye; identifying a plurality of landmarks oneach of the upper eyelid contour and the lower eyelid contour; defininga plurality of triangles, such that each of the plurality of triangleshas vertices formed by a different 3-tuple of landmarks among theplurality of landmarks; computing interior angles of the plurality oftriangles; and computing the eye openness measure of the eye as afunction of the interior angles.
 19. A system for monitoring driveralertness in a vehicle, the system comprising: a camera to capture imagedata representing images of an eye of a driver of the vehicle while thedriver is operating the vehicle; an output device; and a processingsubsystem, coupled to the camera and the output device, to; determine aplurality of thresholds including a plurality of time-based thresholdsand a plurality of angle-based thresholds: receive the image data fromthe camera; compute an eye openness measure of the eye of the driver,based on the image data; determine that the driver is in an unsafe statefor driving based on the eye openness measure, wherein to determinecomprises to apply received or computed data to the plurality ofthresholds; generate an alert, while the driver is operating thevehicle, based on determining that the driver is in the unsafe state;and cause the output device to output to the driver a signal indicativeof the alert.
 20. The system as recited in claim 19, wherein to computean eye openness measure of the eye of the driver comprises to determinean eye state for the eye, and wherein the eye state is from a set of eyestates consisting of: a fully open state, a partially open state and aclosed state.
 21. The system as recited in claim 20, wherein theprocessing subsystem further is to: determine a parameter set based onthe eye state, the parameter set including at least one of a blinkcount, blink rate, eye closure duration, eye open duration, speed ofeyelid closure or speed of eyelid opening; and apply at least oneparameter of the parameter set to a threshold or profile to identify anamount of deviation.
 22. The system as recited in claim 19, wherein: atleast one of the time-based thresholds relates to driver reaction time;at least a first one of the angle-based thresholds relates to the eyeopenness measure; and at least a second one of the angle-basedthresholds relates to a head orientation of the driver.
 23. The systemas recited in claim 19, wherein the processing subsystem further is toupdate the plurality of thresholds while the driver is operating thevehicle.
 24. The system as recited in claim 19, wherein to compute theeye openness measure of the eye of the driver comprises: generating anupper eyelid contour of the eye and a lower eyelid contour of the eyebased on the image data representing images of the eye; identifying aplurality of landmarks on each of the upper eyelid contour and the lowereyelid contour; defining a plurality of triangles, such that each of theplurality of triangles has vertices formed by a different 3-tuple oflandmarks among the plurality of landmarks; computing interior angles ofthe plurality of triangles; and computing the eye openness measure ofthe eye as a function of the interior angles.